Utilizing In-Memory Storage for MPI-IO

In contrast to disk or flash based storage solutions, throughput and latency of in-memory storage promises to be close to the best performance. Kove®’s XPD® offers pooled memory for cluster systems. However, the system does not expose access methods to treat the memory like a traditional parallel file system that offers POSIX or MPI-IO semantics. Contributions of this poster are: 1. Implementation of an MPI-IO (wrapper) driver for the XPD 2. Thorough performance evaluation of the XPD using IOR (MPIO) This MPI independent file driver enables highlevel I/O libraries (HDF5, NetCDF) to utilize the XPD’s pooled memory. APPROACH The developed MPI-IO file drivera is selectable at runtime via LD_PRELOAD. It checks the file name for the prefix “xpd:” and routes the accesses otherwise to the underlying MPI. Important MPI-IO functions for HDF5 and IOR are implemented. During the MPI_open/close the Infiniband connections to the XPD’s are established/destroyed. IOR is used for benchmarking performance and barriers between the phases (open, write, read, close) are used to synchronize the processes. The performance analysis varies the parameters: • Access granularity: 16 KiB, 100 KByteb, 1 MiB, 10 MiB • Processes-per-node (PPN): 1 to 12 • Nodes: 1 to 98 • Connections: 1 to 14 • Access pattern: sequential and randomc • File size: 20 GiB per connection d Performance metrics: M1. Time for open/close (used to test scalability of the connections) M2. Throughput read/write reported by IOR M3. Throughput read/write (computed based on the time for the read/write phase) Each configuration is run at least three times. Since the throughput reported by IOR includes overhead of open/close and initially that times turned out to be significant, those aspects have been investigated separately. A subset of measurements is run on the Lustre of DKRZ’s supercomputer Mistral. ahttp://github.com/JulianKunkel/XPD-MPIIO-driver bBase 10 has been used on purpose as this leads to unaligned access for file systems, i.e., 100 KByte = 105 Bytes. All other cases are base 2. cAs expected for a DRAM based storage system, they did not show significant differences. Thus, the poster only contains values for random I/O. dThe capacity of the XPD is shared among all users. OVERVIEW Performance of all (7500) conducted runs: Fig. 1: Observed throughput computed based on the read/write phase (M3.)